Learning Generative Models with the Up-Propagation Algorithm

Up propagation is an algorithm for inverting and learning neural network generative models Sensory input is processed by inverting a model that generates patterns from hidden variables using top down connections The inversion process is iterative utilizing a negative feedback loop that depends on an error signal propagated by bottom up connections The error signal is also used to learn the generative model from examples The algorithm is benchmarked against principal component analysis in experiments on images of handwritten digits In his doctrine of unconscious inference Helmholtz argued that perceptions are formed by the interaction of bottom up sensory data with top down expectations According to one interpretation of this doctrine perception is a procedure of sequen tial hypothesis testing We propose a new algorithm called up propagation that realizes this interpretation in layered neural networks It uses top down connections to generate hypotheses and bottom up connections to revise them It is important to understand the di erence between up propagation and its an cestor the backpropagation algorithm Backpropagation is a learning algorithm for recognition models As shown in Figure a bottom up connections recognize patterns while top down connections propagate an error signal that is used to learn the recognition model In contrast up propagation is an algorithm for inverting and learning generative models as shown in Figure b Top down connections generate patterns from a set of hidden variables Sensory input is processed by inverting the generative model recovering hidden variables that could have generated the sensory data This operation is called either pattern recognition or pattern analysis depending on the meaning of the hidden variables Inversion of the generative model is done iteratively through a negative feedback loop driven by an error signal from the bottom up connections The error signal is also used for learning the connections